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ABSTRACT 

Reducing spacecraft ground system operations costs is a major goal in all missions. 

The Fast Auroral Snapshot (FAST) flight operations team at the NASA/Goddard 
Spacecraft Flight Center developed in-house scripts and procedures to automate 
monitoring of critical spacecraft functions. The initial staffing profile of 16x7 was 
reduced first to 8x5 and then to ’’lights out”. Operations functions became an offline 
review of system performance and the generation of future science plans for subsequent 
upload to the spacecraft. Lessons learned will be applied to the challenging Triana 
mission, where 24x7 contact with the spacecraft will be necessary at all times. 
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INTRODUCTION 


After a two-year delay due to launch vehicle problems, the Fast Auroral Snapshot 
(FAST) Explorer spacecraft was launched from Vandenburg Air Force Base on August 
21, 1996. Its primary mission was a one-year observation of the Earth’s magnetic field 
above the polar regions. The highly successful mission was extended and continues to 
perform its scientific program today. Ground operations support was provided by NASA 
at the Small Explorer (SMEX) control center facility at the Goddard Space Flight Center 
(GSFC) in Greenbelt, Maryland. Because of budgetary constraints and a desire to transfer 
spacecraft operations to the Principal Investigator (PI) home facility at the University of 
California at Berkeley (UCB), FAST operations personnel developed automation 
concepts and techniques to eliminate the need for man-tended passes and simplifying 
operations as much as possible. 


OPERATIONS CONCEPT 

Because of the volume of science data generated by the FAST complement of 
instruments and the limited amount of on-board data storage, frequent contacts with the 
ground are required. These circumstances drive the requirement to have a ground contact 
every orbit resulting in 9 to 1 1 passes per day. The result is approximately 2 to 3 
Gigabytes (GB) of telemetry downlinked every day. Because most of this data is acquired 
in “real-time” with limited ability to save it, it is important that ground contacts succeed 
in retrieving as much of this data as possible. 

The original operations concept established a 16x7 level of staffing for the flight 
operations team (FOT) to ensure the success of each FAST pass transfer of science data 
to the ground network. This resulted in a staffing level at the FAST mission operations 
center (MOC) of 3 console-level personnel per shift, 2 shifts per day for the life of the 
mission. In addition, a mission planner supported operations 5 days a week on a half-time 
basis. After the critical first six months of the mission when a northern winter science 
campaign was conducted, manned staffing hours were reduced to an 8x5 level. 

After the completion of two years of successful FAST science operations combined with 
budgetary limits at NASA, there was a desire to reduce control center staffing levels as 
much as practical to limit costs while not jeopardizing the science mission. There was 
enough confidence in the spacecraft and ground system to operate the mission with one 
8-hour shift, five days a week with no science commanding occurring over weekends. In 
addition, ground network personnel at the Wallops Flight Facility provided 24x7 anomaly 
support and Barker commanding on weekends. These operations changes resulted in a 
FOT staffing level of 2 console personnel and a part-time mission planner. 

With the approach of the Y2K initiative, a new ground system architecture was 
developed for the FAST control center. This architecture included the introduction of the 
Integrated Test and Operations System (ITOS) for spacecraft command and control 



functions in March 1999. In addition, NASA GSFC’s Spacecraft Emergency Response 
System (SERS) was under development to provide automated spacecraft monitoring 
functions and paging of personnel in the event of anomalous conditions. The combination 
of ITOS and SERS provided an opportunity to automate routine operation functions and 
notify off-duty personnel in the event a spacecraft or ground anomaly which threatened 
the loss of science data or the spacecraft itself. FAST operations milestones and 
corresponding adjustments to FOT staffing are presented in Table 1. 


Table 1. FAST Operations Milestones 


Date 

Operational Milestone 

August 1996-March 1997 

Commence 16x7 manned operations 

March 1997 

Transition to 8x5 manned operations 

March 1999 

Transition to ITOS command and control system 

June 1999 

Start development of lights-out automation concepts 

August/September 1999 

Conduct operations testing of automation operations 

October 1999 

Transition FAST operations to PI facility 


AUTOMATION CONCEPT 


In June 1999, the FAST control center operations team developed a two-phased 
automation concept. The first phase was to automate the activities performed during a 
FAST space-to-ground contact including the sending of commands. The second phase 
was to devise a method to monitor critical spacecraft and payload telemetry and notify 
operations personnel when anomalous conditions were evident. Figure 1 provides a 
system level perspective on the automation concept. 
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Figure 1. FAST System Level Automation Concept 


The automation of routine space-to-ground activities with ITOS was achieved in seven 
steps. First, a schedule-driven approach was adopted. A weekly ground contact schedule 
was ingested into ITOS from the ground network scheduling system. This schedule 
provided ITOS with the acquisition of signal (AOS) and loss-of-signal (LOS) times for 
each ground contact. Second, at 20 minutes prior to AOS, ITOS would setup the pass 
configuration, listen for telemetry and command connection requests from the designated 
ground station, open a new spacecraft events log file for the coming pass, and activate 
mnemonic monitoring scripts for the pass. Third, at AOS, ITOS would determine what 
command activities to perform using a pre-determined, prioritized list of activities. These 
activities were uploading the daily spacecraft functions (e.g., transmitter on/off times, 
science observation timeline, recorder dumps of science and/or spacecraft event message, 
etc.), uplinking the Solar ephemeris files, spacecraft clock adjustment commands, and 
watch dog commands. Fourth, at LOS, after ground station connections were dropped or 
ground station information denoted the end of the spacecraft transmission, ITOS would 
perform close out pass log files and disable monitoring schemes. Fifth, at around LOS+1 
minute, ITOS would produce a pass summary of all the spacecraft event activities that 
had transpired during the pass as well as any problems or failures, i.e., daily load not 
received by the spacecraft for example. These failure notices were sent to the SERS event 
log file as well. Sixth, If a once-a-day activity was successfully concluded during a pass 
(uplink of the daily load file), ITOS would automatically flag this function so it would 
not be performed again for the rest of the operational day. Finally, in the seventh step 
ITOS would notify SERS via the event log file if the pass had a severe anomaly such as 
no telemetry received or having uplinked commands not acknowledged by the spacecraft. 

After the successful automation of routine pass operations, the next step was to automate 
the identification of anomalous conditions and alert operations staff if action needed to be 
taken. This task was accomplished with the use of the SERS. First, operations personnel 
identified key telemetry and command engineering data that were critical to the safety of 
the spacecraft and instruments. If left unchecked, the spacecraft could go into a safehold 
condition or worse. Once these parameters were identified, the staff determined an 
acceptable time interval under which such an anomalous condition could exist before 
corrective action would be required. Next, the type of corrective action was identified for 
each monitored parameter. These actions included logging the event, issuing a page 
and/or sending an email to a designated staff member, or even sending a command to the 
spacecraft autonomously to correct the condition. In the event a page or email was issued, 
if it was not acknowledged within a pre-determined time interval, additional personnel 
would be notified according to a priority list. Operations personnel would receive a brief 
informational message with the page or email to apprise them of the particular problem. 
Additional information was available via an Internet link which provided detailed 
spacecraft logs and telemetry. If immediate action was required, the staff member would 
report to the control center to perform the corrective action. Off-site commanding was not 
permitted. In addition, previous space-to-ground contact information was viewable by 



off-site personnel at any time. Figure 2 provides a graphic representation of the process 
used to implement the automation concepts. 



Figure 2. FAST Automation Concept 
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Transition of the automation concept into operational practice occurred in a very short 
period of time. In August 1999, the ITOS/SERS configuration was activated during the 
manned portion of the operational day. Staff members monitored the system and verified 
























the performance of the system after each spacecraft pass. During various supports, 
individual automated activities were tested. On August 19, 1999, system level testing 
started during operational hours. System testing continued to August 26 th , 1999 when 
“lights-out” operations began 24x7. Full-automated operations continued until the 
operations transitioned to UCB on October 4 th , 1999. 


RESULTS 

During the 40 days of “lights-out” operations, the automation procedures encountered 
only 1 failed attempted at loading a stored command load. The reason for the failed 
attempt was a ground station uplink problem. The stored command load was successfully 
loaded on the following support. 

ITOS and SERS detected three spacecraft anomalies during the 40-day period. The FOT 
was notified of all three anomalies by pager and email. Corrective action took place with 
minimal impacts to operations. 

During the operational use of the automation concept, 100% of spacecraft engineering 
data was recovered along with 99.74% of science data. This reflected the same data 
recovery statistics when FOT supported passes before “lights-out” automation. 

A summary of the activities the FAST automation system performed during September 
1999 is presented in Table 2. 


Table 2. FAST Automation Events and Anomalies for September 1999 


Activity 

Frequency 

Successful 

Attempts 


Stored Command Loads 

Once per day 
between 
Monday-Friday 

29 

l 

Sun Nadir Loads 

Once per day 
between 
Monday-Friday 

29 

0 

Engineering Data Operations 
- Clearing Recorded Data 

Twice per day 

80 

0 

Engineering Data Operations 
- Verifying expected data 
dump, re -dumping if needed 

Every pass 

8 re-dump 
attempts 

0 

Clock Drift Check 


40 


Spacecraft Event Dump 
verification 

Every pass 

15 successful 
event re-dumps 

0 

No-Op Command 

Every pass 

N/A 

N/A 


As evidenced by the table, the system performed extremely well. A total of 520 space-to- 
ground contacts were supported by the automated system. Four anomalous conditions 












were handled without human intervention. As a result of this success, when the flight 
operations responsibility for FAST was transferred to the PI home facility at UCB the 
following month, the automated system was transferred intact. Although the system 
operated at GSFC for a short time, the operations team determined that the new system 
required only one full-time, 8x5 console position to operate the mission. The duties of 
this staff member also included the mission planning functions. 


LESSONS FOR TRIANA 

The Triana spacecraft currently in development will be operated on-orbit in the SMEX 
control center. Operationally it is a demanding mission because it will be transmitting 
telemetry to Earth continuously, 24x7 for 2 years or more. 

The most important lesson learned from the FAST experience was that round-the-clock 
nominal operations could be performed automatically without requiring human 
intervention. Staffing hours for the mission have been aligned with the SMEX concept of 
8x5 operations. Utilizing the FAST automation experience staffing levels for Triana are 
targeted for two full-time personnel. 

Three new approaches have been taken in the development of the automation concepts 
for Triana. The first is that more extensive use will be made of SERS capabilities to 
access insight into spacecraft events and the performance of the ground network used for 
Triana. The Universal Space Network (USN) has installed ITOS at its ground station 
sites. This will enable SERS to determine the results of commanding activities and notify 
operations of problems related to the network as well as the spacecraft. This will be a 
powerful tool for the mission. Second, a formal automation plan is under development for 
Triana. Automation scripts and procedures are being tested and verified with the 
spacecraft today during its integration and test phase. The results of these tests will help 
refine the final procedures put into operations after the spacecraft has reached its 
observation location at Lagrange point 1 and begins nominal mission activities. The third 
approach concerns ITOS. In addition to the Triana MOC and USN using ITOS in the 
telemetry and command flow, the Triana Science and Operations Center (TSOC) located 
at the Scripps Institution of Oceanography in San Diego will be operating an ITOS 
component. It will receive the Triana science from the MOC. The TSOC ITOS will 
provide event logs for access by the MOC SERS, which in turn will alert TSOC 
operations, personnel to system anomalies. This framework of ITOS and SERS 
components throughout the Triana ground system lays the groundwork for powerful 
automation capabilities in the future. 


CONCLUSIONS 

Significant automation of the FAST ground system was achieved in a relatively short 
period of time at little cost. Because of the evolutionary approach of FAST operations 
automation, implementation costs to a total “lights-out” system incurred by the operations 



staff were minimal. There were no additional software costs associated with the 
automation effort. ITOS and SERS software were provided by NASA/GSFC. 

Overall system performance of the FAST ground system exceeded expectations with 
almost no loss of science data while maintaining a healthy spacecraft. The reduction in 
staffing achieved was 50% from the pre-automation levels. With an estimated investment 
of three man-months of work to develop, install and monitor the automation concepts in 
operations, the savings returned in subsequent staffing costs was well worth the effort. In 
addition, the success of the automated system made possible the transfer of FAST 
operations to the PI facility. 

Leveraging the FAST experience for Triana has resulted in realistic staffing plans 
supported by proven automation techniques and the development of more ambitious 
automation schemes and procedures verified with the Triana spacecraft prior to launch. 
Finally, the infusion of software systems and tools throughout the ground system will 
support future advancements in the automation of Triana operations. 
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